reproducible document is a file or set of files, typically in a scientific or data-driven context, that includes both the content (text, tables, figures) & [the code or instructions required to generate and update that content]{uublue-bold}.
Literate programming is a coding & documentation approach where code and explanations are combined in a single document.
Sweave
rmarkdown and knitrQuarto in the both R and Python ecosystemJupyter notebooks using polyglot cellsAdapting the literate programming method to creating documents
The main purpose here is to create documents, slides, websites, blogs, books etc.
The code is a means to creating output for these products, rather than just documenting code
Reproducible documents are a marriage of natural language text (usually q/ some markup) & a scripting language.
| Toolset | Natural language text | Scripting language | Requirement |
|---|---|---|---|
| Sweave | LaTex | R | {{< fa brands r-project >}} |
| Jupyter notebooks | Markdown | Python, other kernels | {{< fa brands python >}} |
| rmarkdown | Markdown | R, but can use Python, shell, others | {{< fa brands r-project >}} |
| Quarto | Markdown | R, Python, Julia, Javascript, others | Independent |
There are many other toolsets as well.
R, Python, and/or anaconda installed on your computer, including the standard packages.Optional content: Either cover quickly or skip, students can review at their pace. As simple as text documents.
Why is markdown important for programmers?README.md files are generally markdownREADME.md files and report generation.ipynb, .qmd, or .rmd[text](URL)..Optional content: Either cover quickly or skip, students can review at their pace.
plain text w/ markup commands to define document structure & formatting.markdown for document generation rather than LaTex. You DON'T have to be a LaTex expert, but, familiarity w/ writing math in LaTex is importantIt’s worth learning a little bit of LaTex both for its mathematical typesetting and for its strong formatting capabilities.
It is common for universities and journals and publishers to provide LaTex document classes for typesetting in their preferred manner.
LaTeX math commands are fundamental for expressing mathematical equations and symbols in documents.
$ $: Enclose mathematical expressions for inline math mode.$$ $$: Enclose mathematical expressions for non-inline math mode.\begin{equation} \end{equation}: Numbered equation w/ label\begin{align} \end{align}: Numbered equation array w/ label\frac{numerator}{denominator}: Display fractions.\sqrt{...}: Add square roots.\sum, \prod: Generate summation and product symbols.\int: Insert integral symbols.\frac{d}{dx}: Represent derivatives.\lim: Display limits.\infty: Represent infinity symbol.\pm, \mp: Add plus-minus and minus-plus symbols.\cdot, \times: Insert multiplication symbols.\frac{\partial}{\partial x}: Indicate partial derivatives.\left(...\right): Automatically size parentheses and brackets.\alpha, \beta, etc.: Use Greek letters.^ and _: Superscripts and subscripts, e.g., x^2 \(\rightarrow x^2\) and a_{ij} \(\rightarrow a_{ij}\).Most reproducible document systems require an installation of LaTex on your computer or system. In particular, both RMarkdown and Quarto use it to generate PDF documents.
There are several distributions of LaTex available, including
However, there is a smaller, essential LaTex distribution available through the tinytex package, based on the TeX Live distribution
quarto install tinytexThis installation remains local to Quarto, i.e., it doesn’t affect other apps on your computer that use TeX. To make this your default TeX installation on your computer, use
quarto install tinytex --update-pathSee the Quarto documentation on PDF Engines, and here for other Quarto tools.
mathpix: https://mathpix.com/
Optional content: Either cover quickly or skip, students can review at their pace.
knitr package, & still has most of its capabilitie
This is still inspired by Sweave
```{r} in line with Markdown syntax
rticles package)Optional content: Either cover quickly or skip, students can review at their pace.
JupyterLab is a web-based interactive user interface
… But
But some would call this a red herring, since you can always to “Run all cells” in a notebook to do things in the right order
Quarto is the next generation of RMarkdown developed by Posit
Quarto is a modern publishing system designed for data science and technical communication, focusing on reproducibility, interactivity, and flexibility:
.qmd chunks)R Language {{< fa brands r-project >}}
.ipynb cells)library(fontawesome)Python {{< fa brands python >}}
| rmarkdown | quarto |
|---|---|
| Requires R | Script-language agnostic (R or Jupyter kernel) |
| Mature ecosystem | New kid on the block |
Uses knitr for weaving |
Uses knitr or jupyter for weaving |
| Various engines for creating final document | Uses pandoc exclusively for document rendering |
| Uses more basic markdown | Allows more complex HTML structures like div blocks, tabsets, callouts, etc. |
xaringan, a popular presentation package based on RMarkdown, renders directly to remark.js and so is not compatible with Quarto
We can create content (text, code, results, graphics) within a source document, and then use different weaving engines to create different document types
Most RMarkdown documents readily render in Quarto
The main story is that there is nothing wrong with either RMarkdown or Jupyter per se, but Quarto enhances the feature set and gives Python a proper reproducible document tool. RMarkdown is still richer.
The major enhancement over both is in creating presentations, and the ease of constructing particular layouts
.qmd files and NOT .ipynb files for the website building.
quarto convert filename commandMarkdown + Code.ipynb stores the code outputs in the meta-data of the file
ONLY HAVE TO RUN THE CODE ONCE with .ipynb.qmd will run the code every time you build the report, which can be very slow
caching options for .qmd to avoid this, however, they are “messier” that just using .ipynb.qmd has no code, then it is basically just a Markdown file .mdTour through various important quarto constructs
- This is an example of a bulleted list with math
- Here is an in-line math equation $f(x)=\frac{e^{x^2}}{2}$
$$ g(x)=x^n \rightarrow \frac{\partial g}{\partial x}=n x^{n-1} $$
$$
\begin{align}
g(x)=x^n \rightarrow \frac{\partial g}{\partial x}=n x^{n-1}\\
h(x)=\int x^n dx \rightarrow h(x)=\frac{x^{n+1}}{n+1}
\end{align}
$$
\[ g(x)=x^n \rightarrow \frac{\partial g}{\partial x}=n x^{n-1} \]
\[ \begin{align} g(x)=x^n \rightarrow \frac{\partial g}{\partial x}=n x^{n-1}\\ h(x)=\int x^n dx \rightarrow h(x)=\frac{x^{n+1}}{n+1} \end{align} \]
::: {layout-ncol=2}::: {layout-ncol=2}
### List One
- Item A
- Item B
### List Two
- Item X
- Item Y
:::Quarto has native support for embedding two popular diagram-generating tools:
```{mermaid}
%%| fig-cap: "A Mermaid diagram"
%%| code-fold: false
%%| output-location: column
flowchart LR
A[Hard edge] --> B(Round edge)
B --> C{Decision}
C --> D[Result one]
C --> E[Result two]
```flowchart LR
A[Hard edge] --> B(Round edge)
B --> C{Decision}
C --> D[Result one]
C --> E[Result two]
.mmd) and Graphviz (.dot) files both in RStudio (using the DiagrammeR package) and in VS Code (using the Quarto extension)::: {layout-ncol=2}
{fig-align="left" width="3.00in"}
{fig-align="left" width="3.00in"}
:::
::: {layout="[[1], [1], [1],[1]]"}
{width=250}
{width=250}
{width=250}
{width=250}
:::
#| layout-ncol: 2
#| fig-cap:
#| - Speed and Stopping Distances of Cars
#| - Vapor Pressure of Mercury as a Function of Temperature
#| code-fold: false
#| vscode: {languageId: r}
#| eval: false
#| echo: true
plot(cars)
plot(pressure)#| layout-ncol: 2
#| label: fig-charts
#| fig-cap: Charts
#| fig-subcap:
#| - Speed and Stopping Distances of Cars
#| - Vapor Pressure of Mercury as a Function of Temperature
#| code-fold: false
#| code-line-numbers: '|3|4-7'
#| vscode: {languageId: r}
#| eval: false
#| echo: true
plot(cars)
plot(pressure).qmd and .ipynb with quarto
quarto convert clustering.qmd this will output a .ipynb version called clustering.ipynbquarto convert eda.ipynb this will output a .qmd version called eda.qmdquarto convert filename.ipynbquarto convert filename.qmdquarto convert filename.rmdquarto preview filename.qmdquarto preview filename.ipynbquarto render filename.qmdquarto render filename.ipynb{{< video https://youtu.be/Z8t4k0Q8e8Y height="400" >}}
[See Quarto Videos for more options and details]
Quarto relies heavily on the YAML format
key: value.- item1, - item2.#.command+control+shift+4 is very useful on a mac.
alt+command+v.
tab is your best friend when using the command line, since it does auto-completionopen ./path_to_file will open any file or directory from the command line –>When creating documents, it is imperative that we properly cite and attribute sources we use in our document
Keeping track of sources and citations often ends up being a big deal
There are several established reference managers available, both commercial and open-source.
BibTeX is a tool and file format that is used to describe and process lists of references. It was developed alongside LaTex.
A typical BibTeX entry looks like this:
@article{article_key,
author = {Peter Adams},
title = {The title of the work},
journal = {The name of the journal},
year = 1993,
number = 2,
pages = {201-213},
month = 7,
note = {An optional note},
volume = 4
}article_key by which you will refer to the citation in your document@article, @book and @unpublishedSeveral reference manages are available that provide citations in BibTeX format, among other formats
We tend to use two free managers: Mendeley and Zotero
.bib file with references that can be used by QuartoTo see some comparisons between these and a popular commercial reference manager, see here
There are a few citation formats that can be used with Quarto, through pandoc.
| Format | File extension |
|---|---|
| BibLaTeX | .bib |
| BibTeX | .bibtex |
| CSL JSON | .json |
| CSL YAML | .yaml |
| RIS | .ris |
Pandoc has the ability to automatically generate citations and a bibliography from BibTeX files and embedded references (much like we replace R/Python code with their output).
Point to .bib files in the YAML header, replacing the name w/ the file-name you’re using
---
bibliography: references.bib
---You can also specify a citation style (e.g., APA, MLA, Chicago) using the Citation Style Language,listed in the Zotero Style Repository
This can be specified in the YAML header as well
---
csl: nature.csl
---You would need to download the .csl file from the repository and keep it in the same folder as your document.
You make a citation using the standard Pandoc/BibTeX syntax ([@citation]), where these are the citation-keys we saw before in the BibTeX entries.
Pandoc will generate a bibliography and place it in the document. It will be placed in a div with the id refs if one exists, otherwise it will be place in the end of the document
### References
::: {#refs}
:::Depending on the CSL format, the bibliography will conform to the ordering and format of that CSL specification.
Citation styles
Optional content: Either cover quickly or skip, students can review at their pace.
Given that Quarto is developed by Posit, the company behind RStudio, Quarto works quite seamlessly in RStudio
RStudio provides both a source editor and a visual editor experience for editing Quarto files
At the top of the file, we put an (optional) YAML header to provide information about document output, general formatting rules and the like.
---
title: "Hello, Quarto"
format: html
editor: visual
------
title: "Hello, Quarto"
format:
html:
theme: cerulean
toc: true
css: styles.css
pdf:
toc: true
number-sections: true
------
title: "Hello, Quarto"
format:
html:
theme: cerulean
toc: true
css: styles.css
pdf:
toc: true
number-sections: true
jupyter: python3
---You specify the Jupyter kernel you want to use as well
Note that YAML is a hierarchical specification with hierarchy denoted by indentation. This is important to maintain, so your specification is clear. Note, for example, that the jupyter option is a top-level option, while the html and pdf specifications are within the format specification
Code is encapsulated in code chunks
#| label: load-packages-r
#| code-fold: false
#| eval: false
#| echo: true
library(tidyverse)You can also add code within the markdown sections, by using r or python syntax
There are `r nrow(ggplot2::mpg)` observations in our data
results in
There are
r nrow(ggplot2::mpg)observations in our data
#| label: load-packages-py
#| code-fold: false
#| eval: false
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as snsTo create “inline” code in Python use python
You can add snippets to the RStudio configuration. Thomas Mock provides several useful Quarto snippets
You can run individual chunks in the R session
You can render Quarto documents from the menu:
or from the {{< fa brands r-project >}} Console
#| eval: false
#| echo: true
#install.packages("quarto")
quarto::quarto_render("hello.qmd")You can use the R Console regardless of whether you’re running a {{< fa brands r-project >}} or {{< fa brands python >}} Quarto document
Quarto in VS Code works much like in RStudio, with some differences.
Run cell button above the code cell
Preview button at the top right of the editor
The basic Quarto document structure is retained in JupyterLab within a ipynb file
But rendering is different
quarto render hello.ipynb --to html
quarto render hello.ipynb --to docxIf you just want a live preview of the notebook (for development), so you can see updates and errors “on the fly”:
quarto preview hello.ipynb --to html.ipynb cells while render the notebook again
--execute flagquarto render hello.ipynb --execute---
execute:
enabled: true
---ipynb notebook and a text-based qmd file
.qmd and .ipynb filesquarto convert hello.ipynb # converts to qmd
quarto convert hello.qmd # converts to ipynb This section is specific to our class setup and workflows and is applicable more generally
We use conda virtual environments in our workflows. These require some special treatment
conda activate anly503quarto render document.qmd --to html
quarto render document.ipynb --to htmlYou can use both {{< fa brands r-project >}} and {{< fa brands python >}} in the same document!!!
knitr engine#| echo: true
#| eval: false
library(reticulate)
# reticulate::use_condaenv('aba688')anly503 conda environment for all Python chunks.
The Jupyter engine does not allow multipe engines to run concurrently, so rendering both R and Python chunks in the same Jupyter document doesn’t work
Sometimes it is useful to extract the actual code from the Quarto document so that it can just run without the overhead of processing the full document or noteboook. This is called tangling
knitr-based quarto documents, one can use knitr::purl#| echo: true
#| eval: false
knitr::purl("mydocument.qmd")will result in creating mydocument.R, where the markdown are placed in comments.
.ipynb if needed) and then do one of the following:
File -> Download as -> Python (.py)jupyter nbconvert --to python notebook.ipynbjupytext to sync the .ipynb file with a .py file (see here)